Bayesian optimization for sparse neural networks with trainable activation functions

In the literature on deep neural networks, there is considerable interest in developing activation functions that can enhance neural network performance. In recent years, there has been renewed scientific interest in proposing activation functions that can be trained throughout the learning process, as they appear to improve network performance, especially by reducing overfitting. In this paper, we propose a trainable activation function whose parameters need to be estimated. A fully Bayesian model is developed to automatically estimate from the learning data both the model weights and activation function parameters. An MCMC-based optimization scheme is developed to build the inference. The proposed method aims to solve the aforementioned problems and improve convergence time by using an efficient sampling scheme that guarantees convergence to the global maximum. The proposed scheme is tested on three datasets with three different CNNs. Promising results demonstrate the usefulness of our proposed approach in improving model accuracy due to the proposed activation function and Bayesian estimation of the parameters

Spatio-temporal wavelet regularization for parallel MRI reconstruction: application to functional MRI

Parallel MRI is a fast imaging technique that enables the acquisition of highly resolved images in space or/and in time. The performance of parallel imaging strongly depends on the reconstruction algorithm, which can proceed either in the original k-space (GRAPPA, SMASH) or in the image domain (SENSE-like methods). To improve the performance of the widely used SENSE algorithm, 2D- or slice-specific regularization in the wavelet domain has been deeply investigated. In this paper, we extend this approach using 3D-wavelet representations in order to handle all slices together and address reconstruction artifacts which propagate across adjacent slices. The gain induced by such extension (3D-Unconstrained Wavelet Regularized -SENSE: 3D-UWR-SENSE) is validated on anatomical image reconstruction where no temporal acquisition is considered. Another important extension accounts for temporal correlations that exist between successive scans in functional MRI (fMRI). In addition to the case of 2D+t acquisition schemes addressed by some other methods like kt-FOCUSS, our approach allows us to deal with 3D+t acquisition schemes which are widely used in neuroimaging. The resulting 3D-UWR-SENSE and 4D-UWR-SENSE reconstruction schemes are fully unsupervised in the sense that all regularization parameters are estimated in the maximum likelihood sense on a reference scan. The gain induced by such extensions is illustrated on both anatomical and functional image reconstruction, and also measured in terms of statistical sensitivity for the 4D-UWR-SENSE approach during a fast event-related fMRI protocol. Our 4D-UWR-SENSE algorithm outperforms the SENSE reconstruction at the subject and group levels (15 subjects) for different contrasts of interest (eg, motor or computation tasks) and using different parallel acceleration factors (R=2 and R=4) on 2x2x3mm3 EPI images.Comment: arXiv admin note: substantial text overlap with arXiv:1103.353

Fast joint detection-estimation of evoked brain activity in event-related fMRI using a variational approach

In standard clinical within-subject analyses of event-related fMRI data, two steps are usually performed separately: detection of brain activity and estimation of the hemodynamic response. Because these two steps are inherently linked, we adopt the so-called region-based Joint Detection-Estimation (JDE) framework that addresses this joint issue using a multivariate inference for detection and estimation. JDE is built by making use of a regional bilinear generative model of the BOLD response and constraining the parameter estimation by physiological priors using temporal and spatial information in a Markovian modeling. In contrast to previous works that use Markov Chain Monte Carlo (MCMC) techniques to approximate the resulting intractable posterior distribution, we recast the JDE into a missing data framework and derive a Variational Expectation-Maximization (VEM) algorithm for its inference. A variational approximation is used to approximate the Markovian model in the unsupervised spatially adaptive JDE inference, which allows fine automatic tuning of spatial regularisation parameters. It follows a new algorithm that exhibits interesting properties compared to the previously used MCMC-based approach. Experiments on artificial and real data show that VEM-JDE is robust to model mis-specification and provides computational gain while maintaining good performance in terms of activation detection and hemodynamic shape recovery

Sparse signal recovery using a Bernoulli generalized Gaussian prior

International audienceBayesian sparse signal recovery has been widely investigated during the last decade due to its ability to automatically estimate regularization parameters. Prior based on mixtures of Bernoulli and continuous distributions have recently been used in a number of recent works to model the target signals , often leading to complicated posteriors. Inference is therefore usually performed using Markov chain Monte Carlo algorithms. In this paper, a Bernoulli-generalized Gaussian distribution is used in a sparse Bayesian regularization framework to promote a two-level flexible sparsity. Since the resulting conditional posterior has a non-differentiable energy function , the inference is conducted using the recently proposed non-smooth Hamiltonian Monte Carlo algorithm. Promising results obtained with synthetic data show the efficiency of the proposed regularization scheme

A hamiltonian Monte Carlo method for non-smooth energy sampling

International audienceEfficient sampling from high-dimensional distribu- tions is a challenging issue that is encountered in many large data recovery problems. In this context, sampling using Hamil- tonian dynamics is one of the recent techniques that have been proposed to exploit the target distribution geometry. Such schemes have clearly been shown to be efficient for multidimensional sam- pling but, rather, are adapted to distributions from the exponential family with smooth energy functions. In this paper, we address the problem of using Hamiltonian dynamics to sample from probabil- ity distributions having non-differentiable energy functions such as those based on the l1 norm. Such distributions are being used intensively in sparse signal and image recovery applications. The technique studied in this paper uses a modified leapfrog transform involving a proximal step. The resulting nonsmooth Hamiltonian Monte Carlo method is tested and validated on a number of exper- iments. Results show its ability to accurately sample according to various multivariate target distributions. The proposed technique is illustrated on synthetic examples and is applied to an image denoising problem

Sparse EEG Source Localization Using Bernoulli Laplacian Priors

International audienceSource localization in electroencephalography has received an increasing amount of interest in the last decade. Solving the underlying ill-posed inverse problem usually requires choosing an appropriate regularization. The usual l2 norm has been considered and provides solutions with low computational complexity. However, in several situations, realistic brain activity is believed to be focused in a few focal areas. In these cases, the l2 norm is known to overestimate the activated spatial areas. One solution to this problem is to promote sparse solutions for instance based on the l1 norm that are easy to handle with optimization techniques. In this paper, we consider the use of an l0 + l1 norm to enforce sparse source activity (by ensuring the solution has few nonzero elements) while regularizing the nonzero amplitudes of the solution. More precisely, the l0 pseudonorm handles the position of the non zero elements while the l1 norm constrains the values of their amplitudes. We use a Bernoulli–Laplace prior to introduce this combined l0 + l1 norm in a Bayesian framework. The proposed Bayesian model is shown to favor sparsity while jointly estimating the model hyperparameters using a Markov chain Monte Carlo sampling technique. We apply the model to both simulated and real EEG data, showing that the proposed method provides better results than the l2 and l1 norms regularizations in the presence of pointwise sources. A comparison with a recent method based on multiple sparse priors is also conducted

A hierarchical sparsity-smoothness Bayesian model for ℓ0 + ℓ1 + ℓ2 regularization

International audienceSparse signal/image recovery is a challenging topic that has captured a great interest during the last decades. To address the ill-posedness of the related inverse problem, regularization is often essential by using appropriate priors that promote the sparsity of the target signal/image. In this context, ℓ0 + ℓ1 regularization has been widely investigated. In this paper, we introduce a new prior accounting simultaneously for both sparsity and smoothness of restored signals. We use a Bernoulli-generalized Gauss-Laplace distribution to perform ℓ0 + ℓ1 + ℓ2 regularization in a Bayesian framework. Our results show the potential of the proposed approach especially in restoring the non-zero coefficients of the signal/image of interest

Solving inverse problems with overcomplete transforms and convex optimization techniques

International audienceMany algorithms have been proposed during the last decade in order to deal with inverse problems. Of particular interest are convex optimization approaches that consist of minimizing a criteria generally composed of two terms: a data fidelity (linked to noise) term and a prior (regularization) term. As image properties are often easier to extract in a transform domain, frame representations may be fruitful. Potential functions are then chosen as priors to fit as well as possible empirical coefficient distributions. As a consequence, the minimization problem can be considered from two viewpoints : a minimization along the coefficients or along the image pixels directly. Some recently proposed iterative optimization algorithms can be easily implemented when the frame representation reduces to an orthonormal basis. Furthermore, it can be noticed that in this particular case, it is equivalent to minimize the criterion in the transform domain or in the image domain. However, much attention should be paid when an overcomplete representation is considered. In that case, there is no longer equivalence between coefficient and image domain minimization. This point will be developed throughout this paper. Moreover, we will discuss how the choice of the transform may influence parameters and operators necessary to implement algorithms